Methods and Apparatus for Vision Enhancement

ABSTRACT

A system and methods for the enhancement of a user&#39;s vision using a head-mounted and user-controllable device including a magnification bubble having variable attributes wherein a portion of the scene is magnified within the complete scene, wherein the user is able to modify, in real-time, how the images are processed including the attributes of the magnification bubble.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional Application No.62/077,434, filed Nov. 10, 2014, U.S. Provisional Application No.62/131,957 filed Mar. 12, 2015, and U.S. Provisional Application No.62/155,972, filed May 1, 2015, U.S. patent application Ser. No.14/937,373 filed Nov. 2, 2015 now U.S. Pat. No. 10,146,304 issued Dec.4, 2018 and U.S. patent application Ser. No. 16/137,003 filed Sep. 22,2018, the contents of which are hereby incorporated by reference intheir entirety.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention generally relates to a vision-enhancement systemand methods, and more particularly, to a head-mounted anduser-controllable method and system for vision-enhancement and a systemand method for configuring a vision-enhancement system.

Discussion of the Background

The normal human visual system establishes a non-magnifiedrepresentation of a scene in the visual periphery with a high-resolutionrepresentation at the center of the visual field. Thus, the cornea andlens of the eye focuses a viewed scene onto the retina, which includesthe fovea near the center of vision and a peripheral area. The fovea isa small area composed of closely packed cones near the center of themacula lutea of the retina. The fovea is responsible for sharp centralvision, which is necessary for activities where visual detail is ofprimary importance, such as reading and driving.

The fovea is greatly expanded at the visual cortex and represents asignificant magnification mechanism that allows a normally sightedperson to discern the region of the visual world that is in “focus” on(faces, titles of dish soap, text), but sees that region in the broadcontext of an overall visual field.

A significant portion of the population suffers from low vision—thosewith visual acuity of 20/80 or worse. These people have difficultyreading, but also with mobile activities such as navigating andrecognizing both people and objects at a distance. This disabilitygreatly diminishes their quality of life, greatly limiting their abilityto socialize, shop, cook and travel. Recent technical advances indigital technology have greatly improved the ability to read with largemagnified CCTV displays, but mobility still remains a difficultchallenge: magnification is effective for static activities like readingor watching TV, but does not enable a person to move through theenvironment while viewing a magnified view of the environment. Thereason is that magnification of the entire image dramatically reducesthe overall field of view. Thus, for instance, if a person with normalvision has a visual field of view of 180 degrees, a system thatmagnifies 6 x reduces that field of view to 30 degrees, thus eliminatinga person's peripheral vision.

Thus there is a need in the art for a method and apparatus that permitsfor enhanced representation of the visual world that also enablesmobility for navigation and recognition. The methods and apparatusshould be easy to use and be inexpensive.

BRIEF SUMMARY OF THE INVENTION

The present invention overcomes the limitations and disadvantages ofprior art vision-enhancement systems and methods by providing the userwith a head-mounted, user-adjustable system that provides a magnifiedview to the user along with peripheral vision.

It is one aspect to provide a device that can present to the user, inreal-time, an enhanced video image of a scene surrounding the user sothat the user can perform routine tasks. Certain embodiments include: 1)a head-mounted video camera that can obtain an input stream of videoimages of the scene in front of the user's head; 2) a processor toprocess the stream of input video images into a stream of output imagesthat corrects for the user's vision deficiencies; and 3) a display forthe processed video images that is viewable by the user.

It is another aspect to provide processed images of a scene thatprovides a smooth transition between an enhanced portion and anunmagnified peripheral portion. Thus, for example, in one embodiment,the processor performs a remapping of the video images to magnify acentral portion of the video images (magnification bubble), to leave theperipheral portion either unmagnified or at a lower power ofmagnification than the bubble, and to provide a smooth transitionbetween the magnification bubble and the peripheral portion so that theuser can easily keep the magnified area within context of the viewedscene.

It is another aspect that the magnification of the scene varies smoothlyacross the image to prevent any discontinuities in the processed imageand to assure that the remapping by itself does not introduce anadditional scotoma or blind area. Thus, for example, the mapping maypresent an unmagnified image that contains no edges and include amagnified image at some central portion.

It is yet another aspect to provide a system that enhances images forpeople with low-vision by mapping each pixel of an imaged scene to anoutput image viewable by the user of the system. The mapping isperformed, in one embodiment, to provide a smoothly varyingmagnification across the output image such that the user of the systemcan easily see how the magnified region is part of the peripheral area.Thus, for example, the user of the system can see every part of theoriginal scene without any visual discontinuities.

It is one aspect to provide a device that can present to the user anenhanced video image of a scene surrounding the user and permit them tomodify the enhancement in real-time. Thus the user can modify, forexample, one or more attributes of the magnification bubble, including,without limitation, the magnifying power, the size of the magnificationbubble, the shape of the magnification bubble, the position of themagnification bubble, and contrast and/or light level of the image.

It is one aspect to provide a portable vision-enhancement systemwearable by a user. The system includes a memory including a storedprogram; a camera mounted on the user aimed to view the scene in frontof the user's head and operable to obtain input video images of thescene; a processor programmed to execute the stored program to transformthe input video images into a stream of output video images whichinclude a magnification bubble wherein the processor is programmed tovary various attributes of the magnification bubble in response to aninput that can be from the user; a screen disposed to be viewed by theuser that displays the output video images; and a user-operablecontroller for generating an input to the processor by which variousattributes of the magnification bubble can be modified.

Certain aspects provide a portable vision-enhancement system wearable onthe head of a user. The system includes a memory including a storedprogram; a camera operable to obtain a stream of input images having afield of view corresponding to the orientation of the head of the userof the portable vision-enhancement system; a processor programmedexecute the stored program to transform the accepted stream of obtainedinput images into a stream of output images; a screen that accepts anddisplays the stream of output images to the eyes of the user; and adevice to accept input from the user. The stream of input imagesincludes a first input image portion and a second input image portionand where the stream of output images includes a first output imageportion including a first magnification of the first input image portionand a second output image portion including a second magnification ofthe second input image portion, where the second magnification isgreater than the first magnification. The processor is adapted to modifythe transform of the accepted stream of input images into the stream ofoutput images according to according to the accepted input. The systemis such that one or more of the eyes of the wearer of the portablevision-enhancement system is presented with an enhanced image of theirfield of view.

Certain other embodiments provide a clinician setup comprising: amonitor; and a controller, where the monitor and the controller are eachconfigured to communicate with the portable vision-enhancement system,such that the monitor mirrors what is displayed on the screen and thecontroller provides access to the stored program, and such that aclinician can view the contents of the screen of the vision-enhancementsystem and, through the controller, modify parameters of the storedprogram such that the clinician controls how the processor modifies thetransform of the accepted stream of input images into the stream ofoutput images.

Certain embodiments provide a method for enhancing vision for a userusing a system including a camera, a processor and a screen. The methodincludes: accepting a user input; accepting a stream of video imagesfrom the camera; image processing the accepted stream of video images inthe processor programmed to include a magnification bubble, where themagnification bubble is varied according to accepted user input; anddisplaying the image processed stream of images on a screen viewable bythe user.

These features, together with the various ancillary provisions andfeatures which will become apparent to those skilled in the art from thefollowing detailed description, are attained by the vision-enhancementsystem and method of the present invention, preferred embodimentsthereof being shown with reference to the accompanying drawings, by wayof example only, wherein:

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING

FIG. 1A shows a first embodiment user-controllable vision-enhancementsystem on a user;

FIG. 1B shows a smartphone used in the system of FIG. 1A;

FIG. 1C shows the body of the goggle used in the system of FIG. 1A;

FIG. 1D shows a second embodiment user-controllable vision-enhancementsystem including devices for tracking eye movement;

FIG. 1E shows a system which may be used to configure avision-enhancement system;

FIG. 2 shows an input image;

FIG. 3 shows a first embodiment output image;

FIG. 4 shows a second embodiment output image; and

FIG. 5 is a flowchart on one embodiment of the transformation performedby smartphone.

FIG. 6 is an illustration depicting the gradual reduction inmagnification between the fully magnified region and the unmagnifiedregion of the scene.

Reference symbols are used in the figures to indicate certaincomponents, aspects or features shown therein, with reference symbolscommon to more than one figure indicating like components, aspects orfeatures shown therein.

DETAILED DESCRIPTION OF THE INVENTION

Certain embodiments of the inventive vision-enhancement system describedherein includes: 1) a video camera to capture the scene that would be inthe user's field of view if the user was not wearing the system; 2) aprocessor to image process the video of the captured scene; and 3) ascreen to present the processed video images to the user's eyes. Themodified video image compensates for the user's particular visionproblem, such as low vision or AMD, by magnifying or otherwise movingcentral image to less impacted portions of the user's eyes whileretaining the peripheral context of scene. Thus, the image of themajority of the scene is presented to the user according to an imagetransformation that maps the captured image, pixel-by-pixel, to an imagethat is more useful to the user.

The image-processed video modifies a scene in a way that allows the userto more easily recognize people or identify items in their surroundings.Thus, the user may be able to identify items on the supermarket shelf,recognize friends and neighbors, read street signs, and sit in anarmchair to read a book.

Certain embodiments of the inventive vision-enhancement system arecontained in a head-mounted apparatus. The head-mounted apparatusgenerally includes a digital video camera with a wide field of view, animage processing system capable of processing the camera imagessufficiently rapidly to provide real-time processed images, and ahigh-resolution digital display, such as an organic light-emitting diodedisplay (OLED) that presents a pair of images, one to each eye of theuser. As described subsequently, the video camera is positioned to viewthe scene in a user's normal field of view and the image-processingimage processes the images for a desired image enhancement. The usersthus see what is presented on the display, which is an enhancement ofthe scene. In certain embodiments, the camera and display are providedby a smartphone that is mounted on goggles.

In certain embodiments, the stream of video camera images is imageprocessed to form a stream of video output images having a magnifiedportion referred to herein as a “magnification bubble.” Themagnification bubble may be central to, or on the edge of, the outputimage. The output image within the magnification bubble is magnifiedrelative to the portion outside of the magnification bubble. Thus, forexample, the output image may be unmagnified, or uniformly magnified,except for a magnification bubble portion which includes a more highlymagnified view of the video camera images. In addition, themagnification varies across the magnification bubble from a maximumvalue within the bubble to the magnification of the surrounding imageoutside of the bubble. The bubble thus has a progressively increasingmagnification from the edge to the center. This can present the imagewith the magnification bubble without image discontinuities so that themagnified portion can be seen within context of the entire image.

By way of a specific embodiment, FIGS. 1A, 1B, and 1C shows a firstembodiment user-controllable vision-enhancement system 100, where FIG.1A shows the system on a user, FIG. 1B shows a smartphone used in thesystem and FIG. 1C shows the body of the goggle used in the system.System 100 includes a smartphone 110 and pair of goggles 120. Smartphone110 includes the electronics necessary for the vision-enhancement system100, including a processor and memory (not shown), a forward facingcamera 111, as shown in FIG. 1A, and a screen 113 on the side oppositethe camera, as shown in FIG. 1B. Smartphone 110 also includes anelectrical connector 117 and may also include a backward facing camera115, which may be used in certain embodiments. As describedsubsequently, processed camera images are displayed on one portion ofscreen 113 shown as a left area 112 and a second portion of the screenis shown as right area 114.

Goggles 120 include a body 122 and a strap 125 for holding the goggleson the user's head and a connector 128 that mates with smartphoneconnector 117. Body 122 includes, as shown in FIG. 1A, a pair of clamps121 for removably restraining smartphone 110 and making the electricalconnection between connectors 117 and 128, and input device 123 forproviding input to the smartphone through the connectors and, as shownin FIG. 1C, a left lens 124 and right lens 126 and a focusing wheel 127.When assembled as in FIG. 1A, with smartphone 110 held in place byclamps 121, system 100 presents what is displayed in area 112 of screen113, through lens 124, to the user's left eye, and what is displayed inarea 114 of the screen, through lens 126, to the user's right eye. Theuser may use focusing wheel 127 to adjust the focus. In certainembodiments, goggles 120 are adapted to accept user input from inputdevice 123, which may control or otherwise provide inputs to theaccepted smartphone 110.

In certain embodiments, smartphone 110 is provided with programming, asthrough a vision-enhancement application (referred to herein as a “VEApp”) which can: 1) operate camera 111 in a video mode to capture astream of “input images”; 2) perform image processing on each inputimage to generate a stream of “output images”; and 3) present the streamof output images to screen 113. In certain embodiments, each of thestream out output images is presented sequentially side-by-side as twoidentical images—one in area 112 and one in area 114. Further, it ispreferred that vision-enhancement system 100 operate so that the timedelay between when the input images are obtained and when the outputimages are provided to screen 113 be as short as possible so that a usermay safely walk and interact with the environment with goggles 120covering their eyes.

In certain embodiments, the VE App may also provide a menu of optionsthat allow for the modification of how vision-enhancement system 100generates an output image from an input image. Thus, for example,vision-enhancement system 100 may execute image-processing algorithmshaving parameters, where the parameters are changeable through the menuby, for example, setting parameter values for magnification, or the sizeand shape of magnification of the output image.

Vision-enhancement system 100 has adjustable features that allow it tomatch the physiology of the user for use in different settings. Thesefeatures are generally set once for each user, possibly with the needfor periodic adjustment. Thus, for example, given the spacing betweenscreen 113 and the eyes of user U, focusing wheel 127 permits for anoptimal setting of the distance to lens 124 and 126. In addition, lens124 and/or 126 may include refractive error correction. Further, it isimportant that the viewed spacing between the images in areas 112 and114 match the user's interpupillary distance (IPD). This may beaccounted for, by example, by shifting the spacing of the output imagesin areas 112 and 114 to match the IPD. Certain embodiments, describedsubsequently, include eye tracking to determine a user's gaze direction.For these systems, it is sometimes necessary to calibrate the system toobtain a correlation between the eye tracking measurement and actualgaze direction.

In various embodiments, the user may adjust setting using: input device123, which may be a touchpad, and which is electrically connected tosmartphone 110, which is further programmed to modify the VE Appaccording to such inputs; a Bluetooth game controller that communicateswith the smartphone 110 via Bluetooth; voice control using themicrophone of the phone; gesture control using available devices such asthe NOD gesture control ring (see, for example,http://techcrunch.com/2014/04/29/nod-bluetooth-gesture-control-ring/);or by the user of an eye tracker to implement gaze-directed control.

In addition, there are other features of vision-enhancement system 100that can either be set up once for a user or may be user-adjustable.These features may include, but are not limited to, adjustments to themagnitude, shape, size, or placement of magnified portions of the outputimage, and color enhancement functions such as contrast, blur, ambientlight level or edge enhancement of the entire image or portions of theimage. In other embodiments, the compass and/or accelerometers withinsmartphone 110 may be used for enhancing orientation, location, orpositioning of output images.

In certain embodiments, sound and/or vibration may be provided onsmartphone 110 to generate for proximity and hazard cues. In otherembodiments, the microphone of smartphone 110 can be used to enter voicecommands to modify the VE App. In certain other embodiments, imagestabilization features or programming of smartphone 110 are used togenerate output images.

In one embodiment, by way of example only, goggles 120 are commerciallyavailable virtual-reality goggles, such as Samsung Gear VR (SamsungElectronics Co. Ltd., Ridgefield Park, N.J.) and smartphone 110 is aGalaxy Note 4 (Samsung Electronics Co. Ltd., Ridgefield Park, N.J.). TheSamsung Gear VR includes a micro USB to provide an electrical connectionto the Galaxy Note 4 and has, as input devices 123, a touch pad andbuttons.

It will be understood by those in the field that vision-enhancementsystem 100 may, instead of including a combination of smartphone andgoggles, be formed from a single device which includes one or morecameras, a processor, display device, and lenses that provide an imageto each eye of the user. In an alternative embodiment, some of thecomponents are head-mounted and the other components are incommunication with the head-mounted components using wired or wirelesscommunication. Thus for example, the screen and, optionally the camera,may be head-mounted, while the processor communicates with the screenand camera using wired or wireless communication.

Further, it will be understood that other combinations of elements mayform the vision-enhancement system 100. Thus, an electronic device whichis not a smartphone but which has a processor, memory, camera, anddisplay may be mounted in goggles 120. Alternatively, some of theelectronic features described as being included in smartphone 110 may beincluded in goggles 120, such as the display or communicationscapabilities. Further, the input control provided by input device 123may be provided by a remote control unit that is in communication withsmartphone 110.

In certain embodiments, vision-enhancement system 100 may includeeye-tracking hardware and/or software, which allow the system todetermine the direction of the user's gaze. Such systems typicallyprocess images of the eye to determine the gaze direction. Thisinformation may then be programmed into smartphone 110 to adjust screen113 such that the transformed image features are automatically shiftedrelative to the gaze. Thus for example, FIG. 1D illustrates avision-enhancement system 100 having eye tracking system 130 including abackwards facing camera 115 that obtains video images of the eye of userU. From these images, software can determine the direction the user islooking and can shift the output images to track the user's gaze.Alternatively, eye tracking may be determined separately from smartphone110, which the tracking information provided to the smartphone. Thus,for example, eye-tracking hardware and/or software may be included ingoggles 120 and communicated through which accepts input by tracking theuser's gaze to a displayed menu.

FIG. 1E illustrates, without limitation, one embodiment of a clinicalsetup 140 that a clinician may user to configure vision-enhancementsystem 100. Clinical setup 140 may allow a clinician to determine andsetup the VE App by setting an IPD, the field of view (fov), backgrounddimming, ambient light level, as well as parameters that are alsouser-adjustable, such as the size, shape, magnification, and location ofenhanced vision features, such as the magnification bubble describedsubsequently.

Clinical setup 140 thus allows for the adjustment or parameters within,or used by, the VE App that smartphone 110 runs to implement thevision-enhancement system 100. Clinical setup 140 includes a monitor142, a Wi-Fi device 144 to allow screen 113 of smartphone 110 to bedisplayed on the monitor, and a Bluetooth controller 146 to communicatevia Bluetooth with smartphone 110. In general, clinical setup 140accepts a video output from smartphone 110 of display 113, and projectswhat the user would see when using vision-enhancement system 110 onmonitor 142.

In certain embodiments, features or aspects of the inventivevision-enhancement system 100 may be adjusted by a clinician usingclinical setup 140. Using the setting up vision-enhancement system 100,screen 113 of smartphone 110 is mirrored on a monitor, using Wi-Fidevice 144, for example, so that the clinician can view what the user isviewing in vision-enhancement system 100. The VE App on smartphone 110includes a menu that allows for the selection of certain parameters thatoperate vision-enhancement system 100.

The clinician has access to the commands in the menu of the VE App viaremote Bluetooth controller 146. In this way, the clinician can “tune”the device to the specific visual demands of the user.

In certain embodiments, Wi-Fi device 144 can be used to remotely add,augment or modify functions allow vision-enhancements, mirror thedisplay, monitor and control VE App configurations in a clinicalenvironment. In certain embodiments, Bluetooth controller 146 can beused to control or modify visual enhancement functions. In certain otherembodiments, the VE App may be reconfigured in a purely magnifiedformat, making it possible for the low vision user to place phone calls,utilize maps, read announcements and perform all visual functionscurrently available to those with normal vision.

Examples of functional control of vision-enhancement system 100 providedby clinical setup 140 through the operation of the VE App may include,but are not limited to:

1. Mirroring the user's display via Wi-Fi.

2. Controlling magnification bubble size and magnification. Theclinician will have the facility to control the low vision parameterssuch as magnification bubble size, magnification, contrast, edgeenhancement ambient level and other visual enhancements to customize thedevice for the user.

One embodiment of the transformation of camera images into a displayedimage is illustrated in FIG. 2, which shows an input image 200, and FIG.3, which shows a first embodiment output image 300. As describedsubsequently, the image transformation accomplishes several objectives.First, a portion of the image is magnified. Second, a portion of theimage surrounding the magnified portion is at a lower magnification, oris not magnified. Third, there is a transition between the two magnifiedportions. This image magnification presents the user with the magnifiedportion of the image within an easily recognizable context of the largerview.

Thus, for example, as shown in FIG. 2, input image 200 represents acamera image view of a scene, which is shown as a grid to aid in thefollowing discussion. In general, input image 200 is any image orportion of an image captured by camera 111 and stored in the memory ofsmartphone 110. FIG. 2 is thus representative of a stored image, whichmay be stored in the memory of smartphone 110 in any one of a number ofvideo formats.

For the purposes of one explanation of the illustrated embodiment, image200 is shown as having an outer portion 210 between an outer edge 201and a first boundary 215, a middle portion 220 between first boundaryand a second boundary 225, and an inner portion 230 within the secondboundary. Boundaries 215/225 are concentric circles, where firstboundary 215 has a radius R1 and second boundary 225 has a radius of R2.

One embodiment performs a transformation or mapping of input image 200into output image 300, as shown in FIG. 3 and which may be, for example,one or both of display areas 112/114. Output mage 300 is shown as havingan outer portion 310 between an outer edge 301 and a first boundary 315,a middle portion 320 between first boundary and a second boundary 325,and an inner portion 330 within the second boundary. Boundaries 315/325are concentric circles, where first boundary 315 has a radius R1 andsecond boundary 325 has a radius of R2′.

The transformation of image 200 to output image 300 maps the imagewithin outer portion 210 to outer portion 310, the image within middleportion 220 to middle portion 320, and the image within outer portion230 to outer portion 330. The transformation generally maps each imagepixel in image 200 a pixel within image 300, within the limitations ofthe image or display resolution. The transformation also includesmultiple levels of magnification, such as high magnification of innerportion 230 a lower, or unity, magnification of outer portion 210, and asmoothly varying magnification in middle portion 220, which ranges fromthe magnification of outer portion 210 at first boundary 215 to themagnification of outer portion 230 at second boundary 215.

Thus, for example: outer portion 210 is mapped to outer portion 310 withunity magnification; inner portion 230, with a radius of R2, is mappedto inner portion 330 with a radius of R2′ at a uniform magnification of(R2′/R2)²; and middle portion 220 is mapped to middle portion 330 from amagnification that varies smoothly from a unity magnification at firstboundary 315 to magnification of (R2′/R2)² at second boundary 325.

The regions of increased magnification, such as regions 320 and 330, arethe magnification bubble. It will be appreciated that the bubble may beround, as in the above embodiments, or may have a generally rectangularshape, which may be useful for reading, or may be placed anywhere withinregion 300, including near edge 301. In certain embodiments, themagnification bubble is stationary on the display, and the user mayposition the bubble by moving their head. The gradual change inmagnification combined with the wide field of view allow a user toeasily position the magnification by their head movement to where it ismost needed. Thus, the user may move their head to magnify words asnecessary and thus easily read the text.

The magnification bubble is thus located within the overall visual scenewith a smooth transition to the unmagnified portion. This allows formagnification of the scene in a way that allows a user to view themagnified image within the context of the overall scene.

The mapping of the image from input image to output image may, inaddition, have certain user-controllable features. Thus, for example andwithout limitation, the magnification is greatest at the center of thebubble and varies gradually and continuously to unity magnification nearthe bubble's edge, and remains constant to the edge of the image. Incertain embodiments, the bubble may be defined mathematically oralgorithmically by several variables which may include, but are notlimited to: 1) the diameter of the bubble; 2) the peak magnification ofthe bubble; and 3) the sharpness of the edge of the bubble (that is, howrapidly the magnification changes near the edge). The user may beprovided with controls (described subsequently) that permit them tochange one or more parameters of the bubble, such as the peakmagnification, the bubble diameter, or the rate of transition ofmagnification from center to edge of the bubble.

Thus, for example, a user may, through input device 123, modify thetransformation to move the magnification bubble within display areas112/114, or change the magnification and/or transition to peakmagnification. FIG. 4, for example, shows a second embodiment outputimage 400, where, relative to output image 300: 1) the region magnifiedand the magnification bubble are shifted to the right, resulting adifferent area to be magnified; 2) the radius of second boundary 225 isdecreased, resulting in a larger magnification with the samemagnification bubble size; and 3) the radius of first boundary 215 isdecreased, resulting in a decreased magnification bubble size and adecreased size of the transition from the unmagnified to magnified imageportions.

In certain embodiments, the bubble may be defined mathematically oralgorithmically by several variables which may include, but are notlimited to: 1) the diameter of the magnification bubble; 2) the peakmagnification of the magnification bubble; and 3) the sharpness of theedge of the magnification bubble (that is, how rapidly the magnificationchanges near the edge). The user may be provided with controls(described subsequently) that permit them to change one or moreparameters of the magnification bubble, such as the peak magnification,the magnification bubble diameter, or the rate of transition ofmagnification from center to edge of the magnification bubble.

In one embodiment, a VE App is provided to smartphone 110 which performsimage processing on each image of the stream of video images and thenpresents each processed area, simultaneously, to areas 112 and 114. FIG.5 is a flowchart on one embodiment of the transformation performed bysmartphone 110.

In Block 1, starts the method, which runs continuously until stopped bythe user.

In Block 2, images are captured of the scene. Images from stream ofimages are read from camera into memory. Depending on the architectureof the particular device and the capabilities and performance of thecamera and the processing CPU, this may be done in one of two ways:

a. Read the entire image into memory and then process all pixels in theimage, or b. Read a collection of pixels (essentially a slice of theimage) and process it while another slide is being read.

In Block 3, Blocks 4-10 are repeated for each pixel in the destination(e.g. output) image

In Block 4, determine if the pixel being analyzed in the output image isinside the magnification bubble. The magnification bubble can be ofvarious shapes. For instance, it may be a circle centered on aparticular point on the screen. It may be an ellipse centered on aparticular point. It may a variety of shapes where the specific shapeand size are personalized to the needs of the user. If the pixel isinside the magnification bubble, perform the steps of Blocks 5-7. If thepixel is not inside the bubble, perform steps of Blocks 8-9.

In Block 5, based on the specific mapping function, determine thecoordinates in the input image from which the particular pixel in theoutput image comes from. These coordinates are referred to as the inputcoordinates.

In Block 6, determine the color values of the pixel from the inputimage. If the input coordinates do not exactly match coordinates of theinput image, estimate that color by using well-known image processingalgorithms such as “nearest neighbor,” “bilinear interpolation,” or“bicubic interpolation.” If the input coordinates fall outside the inputimage, place a fixed value such as black, or some other user-definedvalue.

In Block 7, once the color values have been determined, optionally applyadditional color enhancement functions such as contrast, blur or edgeenhancement.

In Block 8, determine the color values of the pixel from the inputimage. This step is relevant to pixels outside the magnification bubble.The transformation from output coordinates to input coordinates issimpler here and might be as simple as zoom or even no transformation atall. If the input coordinates do not exactly match coordinates of theinput image, estimate that color using well-known image processingalgorithms such as “nearest neighbor,” “bilinear interpolation,” or“bicubic interpolation.” If the input coordinates fall outside the inputimage, place a fixed value such as black or some other user-definedvalue.

In Block 9, once the color values have been determined, optionally applyadditional color enhancement functions such as contrast, blur, or edgeenhancement. This step is relevant to pixels outside the magnificationbubble.

In Block 10, output the resultant color to output image.

In Block 11, repeat this entire process until stopped.

The following is a pseudo code that implements an alternative algorithmfor transforming an input image into an output image. Thus, for example,the following pseudo code may be implemented in software that is storedin a VE App in the memory of smartphone 110. The VE App may beprogrammed to accept input, as through input device 123, for changingone or more parameters of the VE App. Also, in alternative embodiments,the VE App may be operated to provide a menu of whereby the user, or aclinician, may modify one or more parameters in the memory of the VEApp. Thus, for example, VE App may be programmed to accept input, asthrough input device 123, and set a parameter value, either in the VEApp or in a file stored in the memory of smartphone 110, which the VEApp accesses in producing an output image.

The algorithm determines for each pixel of the output image, thecorresponding input image location and color value, and then assigns thecorresponding input image color value to the output image. The followingalgorithm calculates the transform for a circular magnification bubbleat the center of the image with a uniform magnification, Mag, across theinner portion.

For the following discussion, each pixel of the input image (such asinput image 200) at coordinates (x, y), has a color value I(x, y), andeach pixel of the output image (such as output image 300) at coordinates(x′,y′), has a color value O(x′,y′). One embodiment of an algorithmperforms the steps of: 1) for each pixel location of the output image(with coordinates (x′,y′)), determine the mapped (corresponding) pixellocation of the input image (with coordinates (x′, y′)=(x(x′,y′),y(x′,y′)); 2) determine the color value of the input image at the mappedlocation, that is, I(x(x′,y′), y(x′,y′)); and 3) set the color value ofthe corresponding output image pixel to that of the mapped pixel on theinput image, that is, O(x′,y′)=I(x(x′,y′), y(x′,y′)). The mapping(x(x′,y′), y(x′,y′)) accounts for the magnification of the image. Theimage coordinates are further transformed so that the center of themagnification bubble in the input and output images is at(x,y)=(x′,y′)=(0, 0), which may be either at a predetermined imagelocation or maybe input, as by using touchpad 123.

As a more detailed example of this embodiment, a transition parameter kis selected, either using touchpad 123 or by having a preselected value,and a radius R2′ is determined as input, for example, using touchpad123. The value of R1 is calculated from k as R1=R2′*√{square root over(k)}.

For each output image pixel, the distance d from the pixel to the centerof the is calculated as d=√{square root over ((x′)²+(y′)²)}.

Next, the value of d is compared to the values of R1 and R2′. If d>R1,then image area corresponds to the outer portion (portions 210 and 310),and the value of the output image pixel is the same as the value of theinput image at the same pixel locations. That is, the output pixel valueat (x′, y′), which is O(x′,y′), is the same as the input image pixelvalue at the same coordinates (x, y), is the same as the input pixelvalue (x′, y′), and O(x′, y′)=I(x, y).

If d<R2′, then the image area corresponds the inner portion (portions230 and 330), and the image is uniformly magnified by an amount MAG,which may be either preselected or input, for example, using touchpad123. The radius R2′ is mapped to the input image as R2=R2′/MAG, thecoordinates (x′, y′) map to (x/MAG, y/MAG), and thus the output colorvalue is O(x′, y′)=I(x/MAG, y/MAG). For MAG>1, the image within theinput image radius R2 is thus mapped to being within the larger outputimage radius R2′=R2*MAG.

Lastly, if R2′≤d≤R1, then the image area corresponds to the middleportion (portions 220 and 320), and the transformation smoothlytransforms from unity magnification in outer portion 310 to magnifiedinner portion 330. The pixel location (x′, y′) is mapped to:

$\left( {x,y} \right) = \left\lbrack {{\left( \frac{R\; 2^{\prime}}{d} \right)^{2} \times \left( {\frac{1}{MAG} + {\frac{d^{2} - {R\; {2^{\prime}}^{2}}}{R\; {2^{\prime}}^{2}\left( {k - 1} \right)}\left( {k - \frac{1}{MAG}} \right)}} \right\rbrack \left( {x^{\prime},y^{\prime}} \right)},} \right.$

and the output color value is mapped to O(x′, y′)=I(x, y). Thus, forexample, when d=R1=R2′*√{square root over (k)}, the value of

${{\left( \frac{R\; 2^{\prime}}{d} \right)^{2} \times \left( {\frac{1}{MAG} + {\frac{d^{2} - {R\; {2^{\prime}}^{2}}}{R\; {2^{\prime}}^{2}\left( {k - 1} \right)}\left( {k - \frac{1}{MAG}} \right)}} \right\rbrack} = 1},$

the pixel location (x′, y′) is mapped to (x, y), and O(x′, y′)=I(x,y)—the correct value near outer portion 310.

As another example, when d=R2′, and the pixel location (x′, y′) ismapped to (x/MAG, y/MAG), and O(x′, y′)=I(x, y)=I(x/MAG, y/MAG)—thecorrect value near inner portion 330.

In certain embodiments, the user is thus provided with an image thatvaries as they move their head. Thus, for example, it may be most usefulfor a user to have the magnification bubble coincide with the centralpart of the user's vision. In certain embodiments, some calibration ofthe system is required, so that the eye tracking system can adequatelydetermine the location of the gaze.

In one embodiment, the forward looking camera obtains input images of ascene, which is magnified or otherwise transformed from the input imagesand displayed on system 100. If the user wishes to view a modifiedportion of the scene, they move their head to center the area ofinterest in the modified view. In certain other embodiments, eyetracking, as in the embodiment of FIG. 1D, automatically shifts thedisplayed image with the gaze of the user. Thus, for example, eyetracking computes the location of the user's gaze, and shifts the centerof the magnification bubble to correspond to the center of the user'sgaze.

One embodiment of each of the devices and methods described herein is inthe form of a computer program that executes as an app on a smartphone.It will be appreciated by those skilled in the art, embodiments of thepresent invention may be embodied in a special purpose apparatus, suchas a pair of goggles which contain the camera, processor, and screen, orsome combination of elements that are in communication and which,together, operate as the embodiments described.

It will be understood that the steps of methods discussed are performedin one embodiment by an appropriate processor (or processors) of aprocessing (i.e., computer) system, electronic device, or smartphone,executing instructions (code segments) stored in storage. It will alsobe understood that the invention is not limited to any particularimplementation or programming technique and that the invention may beimplemented using any appropriate techniques for implementing thefunctionality described herein. The invention is not limited to anyparticular programming language or operating system.

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure or characteristicdescribed in connection with the embodiment is included in at least oneembodiment of the present invention. Thus, appearances of the phrases“in one embodiment” or “in an embodiment” in various places throughoutthis specification are not necessarily all referring to the sameembodiment. Furthermore, the particular features, structures orcharacteristics may be combined in any suitable manner, as would beapparent to one of ordinary skill in the art from this disclosure, inone or more embodiments.

Similarly, it should be appreciated that in the above description ofexemplary embodiments of the invention, various features of theinvention are sometimes grouped together in a single embodiment, figure,or description thereof for the purpose of streamlining the disclosureand aiding in the understanding of one or more of the various inventiveaspects. This method of disclosure, however, is not to be interpreted asreflecting an intention that the claimed invention requires morefeatures than are expressly recited in each claim. Rather, as thefollowing claims reflect, inventive aspects lie in less than allfeatures of a single foregoing disclosed embodiment. Thus, the claimsfollowing the Detailed Description are hereby expressly incorporatedinto this Detailed Description, with each claim standing on its own as aseparate embodiment of this invention.

Thus, while there has been described what is believed to be thepreferred embodiments of the invention, those skilled in the art willrecognize that other and further modifications may be made theretowithout departing from the spirit of the invention, and it is intendedto claim all such changes and modifications as fall within the scope ofthe invention. For example, any formulas given above are merelyrepresentative of procedures that may be used. Functionality may beadded or deleted from the block diagrams and operations may beinterchanged among functional blocks. Steps may be added or deleted tomethods described within the scope of the present invention.

What is claimed is:
 1. A portable user vision-enhancement systemcomprising: a smartphone including a display, a processor, a memory, anda camera; a pair of goggles wherein the smartphone is operativelymounted on the goggles such that the camera views a scene in front ofthe user and is operable to obtain input video images of a scene; aprogram stored in the memory wherein the processor is programmed toexecute the stored program to transform the input video images of thescene into a stream of output video images of the scene that appear onthe display, where the processor is programmed to vary one or moreattributes of the output video images in response to an input signalprovided by the user to the processor; where said one or more attributesincludes a contrast, a blur, an ambient light level, and an edgeenhancement.
 2. The portable user vision-enhancement system of claim 1,where the one or more attributes includes the contrast.
 3. The portableuser vision-enhancement system of claim 1, where said one or moreattributes includes the blur.
 4. The portable user vision-enhancementsystem of claim 1, where said one or more attributes includes theambient light level.
 5. The portable user vision-enhancement system ofclaim 1, where said one or more attributes includes the edgeenhancement.
 6. The portable user vision-enhancement system of claim 1,where said one or more attributes includes the contrast, the blur, theambient light level, and the edge enhancement.
 7. The portable uservision-enhancement system of claim 1, where said one or more attributesincludes the contrast and the ambient light level.